Corticosteroid treatment prediction using chest X-ray and clinical data

Background and Objective Severe courses of COVID-19 disease can lead to long-term complications. The post-acute phase of COVID-19 refers to the persistent or new symptoms. This problem is becoming more relevant with the increasing number of patients who have contracted COVID-19 and the emergence of new virus variants. In this case, preventive treatment with corticosteroids can be applied. However, not everyone benefits from the treatment, moreover, it can have severe side effects. Currently, no study would analyze who benefits from the treatment. Methods This work introduces a novel approach to the recommendation of Corticosteroid (CS) treatment for patients in the post-acute phase. We have used a novel combination of clinical data, including blood tests, spirometry, and X-ray images from 273 patients. These are very challenging to collect, especially from patients in the post-acute phase of COVID-19. To our knowledge, no similar dataset exists in the literature. Moreover, we have proposed a unique methodology that combines machine learning and deep learning models based on Vision Transformer (ViT) and InceptionNet, preprocessing techniques, and pretraining strategies to deal with the specific characteristics of our data. Results The experiments have proved that combining clinical data with CXR images achieves 8% higher accuracy than independent analysis of CXR images. The proposed method reached 80.0% accuracy (78.7% balanced accuracy) and a ROC-AUC of 0.89. Conclusions The introduced system for CS treatment prediction using our neural network and learning algorithm is unique in this field of research. Here, we have shown the efficiency of using mixed data and proved it on real-world data. The paper also introduces the factors that could be used to predict long-term complications. Additionally, this system was deployed to the hospital environment as a recommendation tool, which admits the clinical application of the proposed methodology.


Introduction
Since 2020, the pandemic SARS-COV-2 (COVID-19) affected the lives of all people in the world.Although some cases have been reported as asymptomatic COVID-19 infections [1], almost 7 million cases (November 2023) of COVID-19 were fatal according to World Health Organization (WHO) statistics.
Despite rapid and continuous improvements in prevention, diagnostics, and treatment, dealing with possible consequences and complications in severe disease courses is still necessary.Those complications (including thromboses, decompensation of comorbid diseases, and fi-Fig.1.The overall scheme of experiments for proposed recommendation system of CS treatment. Currently, there is still a lack of studies that would make personalized recommendations on which patients will benefit from the CS treatment and who will not.Additionally, not so many works use the combined clinical data for analysis because of the difficulty of collecting them.According to WHO statistics, there are 100,000 infections on average a week (at least in November 2023), and there is a higher risk of development of PF after the severe acute phase of COVID-19 [10].
The objective of this paper is to find the relation between a patient's clinical condition in the post-acute phase of COVID-19 at the beginning of treatment and the probability of benefit from CS treatment using clinical information and chest X-ray image of the patient.Sometimes, patients may not be recommended to receive this medication and can avoid possible side effects.
To achieve this goal, we retrospectively analyzed the data from 273 patients showing the results of (non-)application of CS treatment and applied several techniques to preprocess this clinical data.We further proposed an ML-based and DL-based methodology that predicts whether the CS treatment is recommended or not based on the patient's clinical data and the CXR.The application of artificial intelligence (AI) is an appropriate way because of its capability to process a large number of parameters and to find the relations in these data.
The described experiment has two parts: 1. CXR images have been used to predict whether the CS treatment is necessary.This part is performed using our novel deep learning (DL) architecture; 2. CXR images and data retrieved from spirometry, blood information, and anamnesis of the patient's state of health were processed using our DL architecture and ML with a preprocessing of features.
During these experiments, we show that such a combination of information, such as clinical data and CXR images, provides about 8% better accuracy than when it is used separately.The overall scheme of the experiment is depicted in Fig. 1.
The main contributions of this paper are: • Developed DL-based approach for treatment prediction for one of the most frequent long-term COVID-19 complications.The methodology is unique, and there is no comparable one published for this disease yet.• The proposed approach combines traditional machine learning methods for clinical data processing and our architecture of the neural network, which consists of Vision Transformer and InceptionNet, for analysis of CXR images.• This study shows that combining clinical data and CXR images can give more accurate results than when used separately.• A wide range of parameters from patients' anamnesis was analyzed, and the most statistically significant parameters were identified.• The proposed framework was integrated into the internal hospital system as a part of the recommendation tool.• The dataset with clinical data and CXR from patients is introduced and available in [11].
The rest of the paper is structured as follows.Section 2 represents related work and describes relevant solutions based on utilizing mixed data and predicting treatment of other health problems.Next, Section 3 provides more details about the introduced methodology and neural network and the following prediction of CS necessity using clinical and CXR data.Further, Section 4 shows and discusses the experiment results.The last Section 5 concludes the paper.

Related work
This Section represents the latest works and approaches that focus on applying AI in the related research fields to the stated problem.Firstly, the relevant DL methods for CXR processing are introduced.Secondly, the overview of methods for processing mixed data is presented.Lastly, the application of ML for treatment prediction is provided.

Chest X-ray image analysis
CXR and Computed Tomography (CT) are mandatory and the most important diagnostic methods in pulmonology [12].Since the amount of radiation received from a CXR is approximately 70× lower than the radiation received from CT [13], the CXR is still preferred for a general patient's examination due to possible consequent risks.Therefore, the widespread task in the research field is the analysis of CXR images, often performed using DL methods.These methods can be useful as an assistant tool for the radiologist [14].
One of the frequently used ways to construct the neural network for that is to use pre-trained models, such as ResNet [15], DenseNet [16], InceptionNet [17], etc., and to add some new blocks or modules, which help to adapt the model for a given problem.For example, the VGG16 with attention module distinguishes between COVID-19, viral, bacterial pneumonia, and normal [18].Or, the same classes were detected using EfficientNet-B5 with noisy student in [19].Also, the approach [20] utilizes the pre-trained VGG16 model and several augmentation techniques (conventional method, mixup, and Random Image Cropping and Patching (RICAP)) to prevent the overfitting of the model.
One of the first approaches related to COVID-19 detection is described in [21].The authors proposed an ensemble deep learning model consisting of Densenet-121, Resnet-50, InceptionNet, Inception-Resnet, XCeption, and EfficientNet-B2.As the input to the model, four images were generated: cropped and uncropped with sizes of 224 × 224 px and 331 × 331 px.The work results demonstrate that the achieved accuracy of 83% is comparable with the consensus of experienced thoracic radiologists.
The strategy of several branches is also applied in another work [22].The authors used 20 individually trained deep neural networks to detect COVID-19 and non-COVID-19 pneumonia.This ensemble model achieved an AUC of 0.92.The mentioned models can also be modified, as was done in [23].The MobileNet network was adapted here for CXR image classification to avoid the gradient vanishing problem and overfitting.Another work [24] performed experiments for COVID-19 detection using pre-trained models EfficientNetB1, NasNetMobile, and MobileNetV2 and achieved an accuracy of 96%.EfficientNet was also applied for COVID-19 detection in [25].The proposed method has incorporated uncertainty into EfficientNetB3.
Some works describe the models, combining some pre-trained wellknown models, as done in [26].The authors proposed a model consisting of MobileNetV2 and VGG16, and the final classification is performed with the concatenation of these two branches.The authors classified CXR images into COVID-19, Normal, Pneumonia Viral, and Pneumonia Bacterial.InceptionNetV3 can also be applied for this task [27].Additionally to that, CycleGAN was used for data augmentation.Also, a possible way is to use the convolutional network, which consists of convolutional and max pooling layers.According to the authors of work [28], it is possible to achieve an accuracy of 99% for COVID-19 detection.
According to the studied literature, many approaches focus on processing CXR obtained from the acute phase of COVID-19 and detecting viral and bacterial pneumonia and COVID-19.However, there is no attention to the post-acute phase of this disease.

Analysis of mixed medical data
In medical research, data analysis can be done over images, clinical data, or a combination of both.Such a combination of data can give much more information, and consequently, it can increase the accuracy of AI methods [29].Such application is also actual in a pandemic, where predicting the possible complications is necessary.It can assist in prescribing treatment or clinical outcomes and help organize hospital resources.
With this motivation, the authors of work [30] proposed a methodology that uses mixed data to identify patients at risk of severe consequences, which can lead to intensive care or death.The article described three data-processing approaches: handcrafted, hybrid, and end-to-end learning.The best-achieved accuracy is 76.9%, which is produced by a hybrid method.
Another work [31] uses mixed data in an end-to-end manner.The proposed neural network consists of two branches: CNN, which processes CXR images, and MLP, which consists of two dense layers for numerical data processing.The goal is to differentiate between COVID-19 and non-COVID-19 patients.According to the results, the model has achieved 95.4% accuracy.
The comparison of three different scenarios, diagnosis of COVID-19 using clinical data with a fully connected neural network, CXR images with EfficientNetB7, and a combination of both, is introduced in [32].The authors used data from 270 patients.The most accurate case is the combination of clinical data and CXR images, which achieved an accuracy of 97%.
The disease severity and progression are also described in [33].For the experiment, the EfficientNet for image-based severity prediction and a neural network with some dense layers for clinical-data-based severity prediction are used to perform the final severity prediction.Combined progression prediction is performed using a so-called survival forest for output of the dense layer with 256 neurons from the previously mentioned models for images, clinical parameters, and chest CXR severity scores.The results show that the image and clinical data combination performs better for severity prediction (ROC-AUC 0.792) and progression prediction (C-index 0.752) on the external test set.
Another application of DL to mixed data is represented in [34].The methodology consists of the following steps: 1) the 3D CNN extracts the features from CT images, the clinical data are concatenated, 2) dimensionality reduction is performed with the Principal Component Analysis (PCA) method, 3) final classification is performed using CatBoost.The algorithm is aimed to predict categories of patients: who needed intensive care unit admission or were dead and who were healed and moved to non-COVID wards for further care.The achieved results are promising: AUC of 0.949.
Currently, most studies using mixed data focus exclusively on the acute phase of COVID-19 treatment.However, no approach analyzes data from the post-acute phase of COVID-19.In this way, our work fills the gap in research by providing analysis using AI of patients' CXR and clinical data from the post-acute phase.

Treatment recommendation
Another research question is whether it is possible to predict the response to some treatment or an optimal dose of medication using AI.
One of our latest research works [35] is primarily focused on CS treatment prediction, but it is based only on clinical data of patients.This approach compared several ML algorithms, such as Logistic Regression, k-NN, Decision Tree, XGBoost, Random Forest, SVM, MLP, AdaBoost, and LGBM, for the mentioned problem.The best-achieved results belong to the decision tree with a balanced accuracy of 73.52%.Some works have already been focused on this task, but for other fields of medicine.One of the popular fields of research is related to cancer.For example, the work [36] proposes the two-staged DL framework, which consists of tumor segmentation and response prediction.The results show that the segmentation component is essential to response prediction.
Another approach combines the information from the CT image and clinical data [37].The article retrospectively analyzes the outcome predictions for individualized radiotherapy doses.The proposed model predicts treatment failures with a C-index of 0.72.Also, ML was applied to predict insufficient response to methotrexate in patients with Rheumatoid Arthritis [38].The results show logistic regression is the most successful in this task, and achieved an AUC of 0.77.On the other hand, some works focused on predicting the outcome of treatment for COVID-19 [39].For example, work [40] proposes a DL framework with Whale Optimization Algorithm (WOA), which utilizes CT images and some additional information about patients (age, infection stage, etc.) and is aimed to predict the patient's response to treatment during the acute phase.

Related work summary
All approaches mentioned in this section are summed up in Table 1.Considering the studied literature and the current state of technologies, it can be concluded that there is no work focusing on retrospective analysis based on clinical data and CXR images of patients' COVID-19 post-acute phase.Additionally, no ML/DL approach would recommend treatment for one of the most severe long-term post-COVID complications, except our work.This way, we aim to fill the gap in the research field because our work represents the analysis of such data and provides a unique dataset with a combination of the clinical data and CXR images and a detailed statistical evaluation of the proposed methodology.

Methodology
This section introduces a DL-based approach to recommend preventive CS treatment for one of the most severe long-term post-COVID-19 complications -PF.
First, data are preprocessed, including image processing and selection from the clinical data.Then, the recently proposed our neural network and the following comparison with other architectures are described.Finally the two scenarios of experiments: 1) with only CXR images, and 2) a combination of CXR and clinical data, are introduced.
Generally, the experiment consists of the following steps (see Fig. 1): 1. Training our neural network on open large dataset ChestX-ray14 [41] and comparing it with other well-known architectures to select the most successful architecture for further use; 2. Testing trained model selected from the previous step on given CXR images to determine the necessity of CS (experiment 1); 3. Statistical evaluation and selection of most significant parameters from clinical data; 4. Application of DL and ML to combined data: CXR image and selected parameters from clinical data (experiment 2).
The dataset [11] and source code [42] are available online.

Data preparation
One of the main parts of ML methods is data preparation.Generally, this phase consists of CXR image preprocessing, parameter selection, and labeling.

Custom dataset
In this work, the new dataset with CXR images and clinical data of patients from post-acute COVID-19 is introduced and provided by University Hospital Olomouc, Czech Republic.Patients' data were collected during the initial check-up, which is 4 -12 weeks after the acute onset of COVID-19.All of the examinations, including blood tests, CXR, and pulmonary function tests, were performed at the same time.
The dataset has a total of 273 patients, where the information about the necessity of CS is given: 141 patients will not benefit from CS treatment, and 132 will benefit from it.It is essential to note that all of the patients requiring CS treatment had pneumonia during the acute phase of COVID-19.In addition, all of those patients had persisting pulmonary involvement.However, in a significant proportion of the patients, there was spontaneous regression of the lung damage.The model aimed to identify those with profit from CS therapy.This dataset includes more than 100 parameters, including blood test values, pulmonary function test (spirometry), the patient's anamnesis, clinical condition, and X-ray images as summarized in Table 2.
Notably, several patients had negative or N/A results for COVID-19 testing -anti-SARS-CoV-2 IgG.COVID-19 was diagnosed with very suspicious clinical symptoms, radiology images, and the presence of anti-SARS-CoV-2 IgM antibodies (only in non-vaccinated individuals at the beginning of the epidemic).

Parameters selection
The parameter selection from the clinical data represented with quantitative and categorical numbers is essential for avoiding overfitting.The clinical data consist of information taken at the time of initiation of post-COVID treatment.In this step, the parameters are pre- selected based on two statistical methods and manual selection.The feature selection is depicted in Fig. 3.The first one is SHapley Additive exPlanations (SHAP) [43], which explains the feature importance for a given ML method, i.e., the Decision Tree in our case.The output is depicted in Fig. 2. As can be seen, the amount of CS received during the treatment (i.e., dose × period) (CS_amount) and Immunoglobulin M antibodies (IgM) (SARS-CoV-2 IgM(quant.))values from blood tests significantly impact CS treatment.IgM indicates an early immune response after infection in the body.
Additionally, some other parameters were pre-selected manually after series of trials to achieve the best effect of CS prediction.These parameters correlate with the severity of the pneumonia and impact the persisting pulmonary damage, leading to an indication of CS treatment.A brief description of selected ones is introduced below: (for more than 4 weeks); • Persistent dyspnea -presence of long-lasting dyspnea after COVID-19 (for more than 4 weeks); • MEF25 (abs.)-maximal expiratory flow at 25% of vital capacity.

Data labeling
The clinical data also contains information about the objective radiological score, which reflects the lung damage regression in the range of 0 (immutable state) to 10 (complete regression).This parameter gives information on whether the patient's health has improved.Patients with scores between 0 and 6 are not considered to have significantly improved their health status.For the values 7-10, it is considered that their status improved.Considering this information, it was possible to determine the necessity of CS, i.e., whether the patient will benefit from the CS treatment.The dataset was split into training and testing subsets.

Chest X-ray image preparation
The CXR images were cropped to the area of interest for all experiments.This step uses a pre-trained U-Net [45] model for lung segmentation, used in [21].This step helps the neural network to focus on the lungs instead of considering the surrounding area.
Another important step is data augmentation to reduce the model's possible overfitting.For this purpose, transformations (e.g., translation, rotation, and zoom) were applied to extend the dataset during the training.
This work uses two datasets with CXR images: the ChestX-ray14 dataset [41] for pre-training and our custom dataset for CS prediction.
ChestX-ray14 has a total of 112,120 images with 14 pulmonary diseases: Atelectasis, Cardiomegaly, Consolidation, Edema, Effusion, Emphysema, Fibrosis, Hernia, Infiltration, Mass, Nodule, Pleural thickening, Pneumonia, and Pneumothorax.Among the mentioned diseases, some images do not have any disease.It is one of the most frequently used datasets in this field of research.However, the images with at least one pulmonary disease were selected for this experiment.The dataset was divided into training (36,024 images) and testing (15,735 images) parts according to the official dataset split.Also, the validation set is 20% of the training set.

Neural network for feature extraction from X-ray images
A neural network is selected for image processing in this part of the experiment.First, the description of our architecture of the neural network [46] is presented.Then, a comparison with other well-known architectures is performed.

Architecture
The recently proposed neural network is aimed to perform the classification task over CXR images.The first step is to classify images from the ChestX-ray14 dataset, which is prepared for multilabel classification.That is why the following description will be focused on multilabel classification.The overall scheme is introduced in Fig. 5.
This architecture applies the strategy of two branches to capture global and local features.This concept has been widely used in literature [26,47,48].On the one hand, using only one branch is not so effective, according to the results presented in the following sections.On the other hand, applying more branches increases the model's complexity, making it more computationally demanding.That is why the optimal solution in this case is using two branches.The first branch aims to extract the global features and is represented with InceptionNetV3 [17].After InceptionNetV3, GlobalMaxPooling, Dense, and Dropout layers are used for the final classification.The second branch consists of four parts.First, feature extraction is performed with three inception modules.Fig. 4 shows the architecture of the inception module.After the inception modules, the Vision Transformer (ViT) [49] was utilized for the following processing.Fig. 6 shows the architecture of ViT in more detail.
The concatenation of two branches consists of three dense layers with 64, 32, and 14 units to provide the final results.The last layer  Considering that the dataset is slightly imbalanced, the Asymmetric Loss (ASL) function was used [50].Additionally, it was modified: instead of binary cross-entropy, the focal loss function was applied.Mathematically, it has a unified formula (more details can be found in the original paper) [50]: where  + and  − are the positive and negative loss parts,  is the output probability from the network,   is the shifted probability,  is the focusing parameter.The set up parameters are  − is 5,  + is 1, clip is 0.001.
The focal loss function is described in [51]: where   is the model's estimated probability for the class with label  = 1,   is the balancing factor, and  is the modulating factor.

Comparison with traditional methods
To compare the proposed model with other methods, some wellknown architectures for image classification were selected: DenseNet121 [16], EfficientNetB4 [52], InceptionNetV3 [17], ResNet101 [15] and VGG16 [53].These models were also trained on the ChestX-ray14 dataset to detect diseases in CXR.The following values were used for hyper-parameter tuning for all the mentioned architectures, including our architecture: batch size -8, number of steps per epoch -1000, num- ber of epochs -100, optimizer -AdamW, learning rate -0.0001, weight decay -0.00001.All mentioned methods used as a baseline apply the binary cross-entropy as a loss function.Tables 3 and 4 represent the results of trained models.The perfect and maximum value for each metric is 1.Based on the results, it can be concluded that the proposed model achieves better accuracy than other neural network architectures with AUC -0.7952, specificity -0.7326, accuracy -0.7334, precision -0.2541, and F1 score -0.3478.The EfficientNetB4 model has worse results than the proposed model but has achieved better results among other methods from the baseline.It has an AUC of 0.7772, sensitivity of 0.7079, specificity of 0.7221, accuracy of 0.7223, precision of 0.2419, and F1 score of 0.3324.On the other hand, the worst results were achieved by ResNet101 for such metrics as accuracy (0.7520) and sensitivity (0.7011).Tables 3 and 4 also show the results for InceptionNetV3, used in the proposed model.It performed with an AUC of 0.7522, sensitivity of 0.7073, specificity of 0.6869, accuracy of 0.6864, precision of 0.2314, and F1 score of 0.3171.As can be seen, the application of InceptionNetV3 independently has worse results for all metrics than the proposed model.From this fact, it can be concluded that the application of ViT and asymmetric loss function is effective for this field of research.
Considering the above-mentioned results, our model is the most suitable for the following experiments.

Experiments with custom dataset
This section describes the experiments on our custom dataset containing CXR images and clinical data.As mentioned in previous sections, the experiment's goal is to determine whether treating the patient with CS is recommended.This experiment is divided into two parts.The first one is aimed at experimenting only on CXR images.The second part is designed to apply CXR images and clinical data.The pre-trained model from section 3.2 is used for both parts.

Classification of chest X-ray images
This part of the experiment is supposed to determine whether the CS treatment is needed, taking into consideration only a CXR image of the patient.The feature extractor is the main part, which is a pre-trained neural network from the previous step (see subsection 3.2.1).Instead of adding new layers to a pre-trained network, which would perform the binary classification, we applied traditional ML methods to the latent space of the three last layers of the neural network.The combination of NN and traditional ML has been successfully applied in some other works, for example, in [54,34,55].This method was selected because fine-tuning pre-trained NN and adding several new layers to perform the classification can be ineffective since the dataset is relatively small.The traditional ML algorithms in our experiment were independently applied to the last three layers.Consequently, three different results were achieved.The scheme of the experiment is depicted in Fig. 7.
An optimization technique, random search with cross-validation with five folds, is used to find the best hyper-parameter combination.The found optimal hyper-parameters for different ML methods and different outputs from the last layers are shown in Table 5.

Classification with mixed data
The second part of the experiment aims to determine the necessity of CS, considering the combination of CXR images and clinical data.The experiment scheme is depicted in Fig. 8.
As can be seen, the CXR image is fed into our neural network, which is pre-trained on general pulmonary diseases, where the features are extracted.The outputs from the last layers are concatenated with the preselected clinical features (see section 3.1.2).Here, it is considered that additional information can improve the prediction accuracy of the necessity of CS treatment.And, as was described in the previous part, the traditional ML methods are applied independently to each pair layer output + clinical data.There were three experiments because the three last layers were used for the experiment.
As was presented in the previous part, the set of ML methods is used the same.Here also the optimization technique, such as random search with cross-validation ( = 5).The optimal hyper-parameters are found and introduced in Table 6.

Results and discussion
This section represents the metrics used for comparing trained models, discusses the achieved results, and the future directions for further research.

Metrics
The metrics used are accuracy, precision, recall, balanced accuracy, and F1-score, which are usually used in evaluating classification tasks.The definitions of the mentioned metrics are shown below [35]: =     +   , (6) where   -number of True Negative cases,   -True Positives,   -False Positives,   -False Negatives.

Classification of chest X-ray data
This section introduces the results for predicting CS treatment necessity based on CXR images.Table 7 shows the results for different last layers output processed with some traditional ML methods.The best potential result for all metrics is 1, and the worst is 0. As can be seen, the most successful method is the RF, which was applied for the last layer (-3).The results are: accuracy is 0.7273, F1 is 0.7059, balanced accuracy is 0.7267, and ROC-AUC is 0.7247.However, MLP has the best result for recall 0.76.The kNN method from the last layer (-1) has the best precision result 0.73.
For the other layers, the results are close to the ones mentioned above.For the last layer (-1), RF has the best results for accuracy (0.7091), F1 (0.6923), recall (0.72), balanced accuracy (0.71), and ROC-AUC (0.71).The RF performs better than other methods for the last layer (-2).It succeeded with an accuracy of 0.7091, F1 of 0.6522, precision of 0.7143, balanced accuracy of 0.70, and a ROC-AUC of 0.6933.DT method is the best for recall (0.64).
Another interesting direction for investigation is the identification of the disease that has the most significant impact on the decision for CS treatment.As mentioned above, for this experiment, the used neural network was pre-trained on the large dataset ChestX-ray14, which is labeled for 14 different pulmonary diseases.The last layer output of this Fig. 9. SHAP evaluation for the outputs of last layer (-1) of NN in experiments with only CXR. model is a vector of probabilities of various illnesses.This output is fed into selected traditional ML algorithms, and the decision on CS treatment is performed.To investigate, which values of NN output are the most significant for decision-making, the SHAP analysis was applied to explain the model.Fig. 9 shows the feature importance for the selected algorithm, in this case, for RF (accuracy is 0.7091).As can be seen, the highest values have the following diseases: Effusion, Fibrosis, and Edema.The other ones have less impact on output, but it is still high.

Classification of mixed data
Another conducted experiment has not only CXR image data but also additional clinical information about every patient (blood test, spirometry, and anamnesis).The results are shown in Table 8.Here, it is more difficult to determine the absolutely successful combination of one of the last layers and the ML method.
However, the MLP for the last layer (-2) has the best results for accuracy (0.80), balanced accuracy (0.7867) and ROC-AUC (0.8933).On the other hand, DT for the last layer (-1) has achieved the best value ment and not recommend it to patients who, with a high probability, will not benefit from it and in this way, overtreatment will be avoided.We also conclude that combining the raw CXR image data with clinical data is significantly more accurate than just CXR images.Despite the application of relatively computation-demanding architectures, such as InceptionNet and Vision Transformer, the proposed architecture can be applied to real-world environments.This is proved by deploying the methodology into the internal hospital system.This work attempted to use the optimal input image size so that the details would be preserved, and it is not computationally demanding.On the other hand, if this methodology is applied in another hospital environment, it is necessary to consider the allowed latency, processing time, and frequency of usage.Depending on these points, the hardware should be chosen wisely.However, this work has some issues and requires some improvements.
Firstly, the relatively small custom dataset is used, which is why the overfitting problem can appear.As future work, this dataset is supposed to be extended, especially with the appearance of new mutations.Here it is also worth mentioning, that some patients were diagnosed with other methods, than anti-SARS-CoV-2 IgG testing, but with clinical symptoms, radiology image, and presence of anti-SARS-CoV-2 IgM antibodies.It indicates, that other clinical information should also be considered during patient examination and treatment prescription.
Secondly, in this work, the model was pre-trained on general pulmonary diseases.However, in future work, the model should be pretrained also on COVID-19 datasets.It would make the NN more adapted to the problems related to COVID-19.
Thirdly, this paper utilizes the two methods of feature selection.However, the following work can extend this part too: more detailed analysis of clinical data using statistical methods and different parameter selection techniques is required.One of the possible directions is to use feature selection based on genetic algorithms as was proposed in [57].
Another raised problem is the interpretation of algorithm decisionmaking.Since the neural network is a kind of "black-box" it is necessary to apply the advanced interpretation techniques of such methods.This point will be addressed in future work.
Considering the complexity of ML/DL methods, it is reasonable to experiment with other techniques, for example, fuzzy similarities [58,59].Lastly, based on the received information on the patient and his treatment, the work can be extended by evaluating the severity and duration of the post-acute phase.

Conclusion
Prevention, diagnostics, and treatment of COVID-19 have improved significantly in recent years.However, severe cases may lead to longterm complications that may increase worldwide morbidity.One of the most severe complications is PF, which can be preventively treated using CS.Unfortunately, CS treatment has severe side effects, and the non-selected application of CS is considered harmful.Currently, no methodology can identify people who will benefit from the treatment and who will not.
This work introduced a novel, personalized AI-based methodology for CS treatment prescription during the post-acute phase of COVID-19.For the design of the methodology, we collected data from a total of 273 patients, which consist of more than 100 different parameters for each person.It includes data that were collected at the first examination of post-COVID treatment (blood tests, spirometry test, anamnesis, and CXR images), which is, regarding its detail, the largest reported dataset.
The proposed methodology is based on a neural network architecture pre-trained on a large dataset of pulmonary diseases ChestX-ray14.In this way, the neural network can extract the relevant features from input images.The first experiment is based on the processing of a CXR image by NN and the traditional ML algorithm with optimization, which was applied to get the final decision on the necessity of CS treatment.
The second experiment has the additional part: clinical data concatenated to the NN output for the final decision.
Here results from CXR and a combination of CXR with clinical information were evaluated.We demonstrated that combining CXR and clinical information achieves significantly better results (78.7%).
The proposed methodology can be used in real-world practice.It can assist physicians in identifying individuals who need CS treatment.For sure, this field of research is promising.It can help improve the speed of the patient's examination, and more people would get personalized medical care and avoid overtreatment of patients who will not benefit from the treatment.The same principle may be applied to other diseases with a risk of PF development (e.g., organizing pneumonia, acute respiratory distress syndrome, or drug-induced lung involvement).This way, the pandemic of COVID-19 may give us lessons for future applications of AI.

Fig. 6 .
Fig. 6.Scheme of the transformer network used in the proposed architecture.

Fig. 7 .
Fig. 7. Overview of the experiment with fine-tuning and application of ML methods.

Fig. 8 .
Fig. 8. Overview of the experiment with a combination of CXR and clinical data.

Table 1
Comparison of related works.

Table 2
Description of patients' clinical data from the custom dataset.

Table 3
Evaluation of the achieved results by all methods.

Table 4
Detailed AUC results achieved with all methods for each disease.

Table 5
Detected hyper-parameters of machine learning methods for experiments with only CXR.

Table 6
Detected hyper-parameters of machine learning methods for experiments with CXR images and clinical data.